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Abstract 

The purpose of this study was to determine if competently reasoning clinicians could: (1) 
identify reasons competently reasoning students used to eliminate distractors; and, (2) explain 
how students would reason to select the keyed response when solving multiple-choice items. 
Clinicians (n=5 to 11) think aloud protocols for four multiple-choice items were analyzed by 
categorizing and coding the clinicians’ explanations, and compared to students’ (n=18 to 32) 
justifications. 

The results showed that: (1) expert clinicians were more successful in identifying the 
students’ justifications for eliminating distractors than novice clinicians; and, (2) novice clinicians 
were more successful than expert clinicians in explaining how students reasoned to select the 
keyed response. These results suggest that the expert and novice clinicians’ decisions 
complimented each other in identifying the distractor elimination and perceptions of how 
students solved items. 
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CLINICIANS’ EXPLANATIONS OF STUDENT REASONING 
WHILE SOLVING MULTIPLE-CHOICE ITEMS 

O.H. Triska 1 , T.O. Maguire 1,2 , C.C. Harley 2 . 

Faculty of Education 1 , and Faculty of Medicine 2 , University of Alberta, Edmonton Alberta, CA 



Introduction 

To a large extent the quality of multiple-choice (MC) items depends upon the ability of 
clinical instructors who create the items to place themselves in the role of students, and create 
distractors that represent the kinds of misconceptions that students are likely to have. Since 
clinicians are further along the continuum of professional knowledge than students, it may be 
difficult for them to move back along the continuum to reason at a student level. The purpose 
of this study was to determine if competently reasoning clinicians could: (1) identify reasons 
competently reasoning students used to eliminate distractors in MC items, and, (2) explain how 
students would reason to select the keyed response when solving MC items. For this study, 
competence was defined as those individuals who chose the keyed response. 

The theoretical foundation underlying this study was an information processing theory 
proposed by John Anderson (1983; 1993), the ACT* theory. According to Anderson (1993), 
cognitive processes used by problem solvers can be viewed as a sequence of internal states of 
knowledge successively being transformed from declarative knowledge processing and deeper 
problem representation through to practice and expert problem solving skills. Declarative 
knowledge becomes very large and well organized while the procedural knowledge becomes more 
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specialized and efficient. 

Anderson (1983, 1993, 1994) explained that declarative knowledge and procedural 
knowledge are distinct. The initial stages of cognitive skill development are termed the 
interpretive stages. Declarative knowledge is conceptual and factual and is the result of acquiring 
information about something. Novices use declarative knowledge with general procedural 
knowledge to solve domain-related problems. Eventually, the declarative knowledge becomes 
elaborate and well organized. However, as the complexity of problems increases, general 
knowledge is insufficient to reach a solution. When individuals reach a state where there are no 
adequate solutions, they will search for an example of a similar problem-solving situation and try 
to solve the problem by analogy to that example. Specific scenarios are recalled and interpreted. 
Knowledge is retrieved from declarative memory. Individuals rehearse pivotal issues from the 
situation and substantial verbalization can occur. As the verbalization lessens, the cognitive skills 
are encoded procedurally. This transition is termed knowledge compilation and is the associative 
link between the interpretive stage and autonomous stage of procedural cognitive skill 
acquisition. 

The cognitive skill assessment model proposed by Glaser, Lesgold, and Lajoie (1985) 
functioned in concert with the ACT* theory (Anderson, 1983; 1993) to provide a six dimension 
continuum for determining levels of competent reasoning in medical examinations. As noted 
earlier, assessment of competency requires clinician instructors to project how far along the 
continuum of medical education the student should be. This task requires clinician instructors to 
assess the level of clinical reasoning skills of the medical students about to enter residency. 

These skills tend to be sequential in a domain-specific area of knowledge and span several 
dimensions, from knowledge differences of novices to experts, with intermediate stages, and 
transitions from level to level. First, beginners’ knowledge is fragmented. It consists of isolated 
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facts, definitions, and a superficial understanding of the domain vocabulary. At the first level of 
assessment, knowledge organization and structure, can be examined to determine the elements 
and components in a domain of knowledge. The objective is to determine the interconnections 
between the chunks of information and fragments of knowledge. 

Second, one can assess the depth of problem representation an individual possesses. Novices 
typically recognize the surface features of a problem or task. More proficient individuals identify 
inferences, principles, and concepts imbedded below the surface structure. Individuals who 
immediately recognize underlying principles solve problems rapidly by spending little time on 
details. They arrive at the correct answer quickly. 

Third, the quality of mental model can be investigated. Individuals develop mental models 
of scenarios and situations consistent with their domains of knowledge. The nature of these 
representations is determined by the tasks necessary to execute the performance of skills at a 
specific level. As tasks become more complex, the model is amended to incorporate these new 
skills. The mental model not only indicates the level of task complexity but also the level of 
cognitive ability needed to solve the problem. 

Fourth, the efficiency of carrying out procedures can be looked at. As individuals’ mental 
models become more refined, tasks are performed more effectively. Well-practiced procedures 
are important for understanding and comprehension of a scenario. At this level, assessment 
should focus of the relationship between understanding the requirements on a task and 
performing the task. 

Fifth, automaticity should be present. At higher levels of cognitive development, previously 
learned basic skills are practiced enough so that they become automatized and are performed 
with little conscious attention. When performing complex tasks, individuals’ attentional demands 
are taxed simultaneously. As this occurs, the efficiency of the overall task is compromised. The 
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criterion for assessment is whether the automaticity of the basic process has progressed to a point 
where the subtasks have minimal interference on the total performance. 

Sixth, metacognitive activities should be present. At this level, individuals monitor their 
performance and identify strategies for solving problems using procedural skills. Individuals 
reflect on and control their performance. They know what they know or don’t know. They can 
predict the outcome of their performance. They can plan their activities in advance and 
efficiently apportion their time. They can check and monitor the outcomes of their problem 
solutions and attempts to learn. These regulatory skills develop with experience and practice. 
Evaluating metacognitive skills could be an important facet of predicting problem-solving abilities 
that result in learning. 

Glaser et al., (1985), explained that these skills tend to be indexed in a domain of 
knowledge. Because these skills are not discreet but on a continuum, identifying assessment tools 
appropriate for each level is difficult, particularly at the transitions points and at higher levels of 
cognitive development. 

Differentiating among the higher levels of knowledge is demanding because experts’ 
cognitive structures are well developed. The speed of processing knowledge is so fast that the 
borders among levels are indistinct, particularly between efficiency of procedures and 
automaticity. Metacognitive activity can be difficult to identify when individuals change their 
train of thought, for example when they generate a hypothesis, then consider peripheral issues 
and offer another solution to a problem. In responding to a MC item, experts might read an 
item stem and generate a hypothesis. If after thinking further about the scenario, they change 
the hypothesis? Did the experts just change their minds, or did they reconsider the information 
presented in the stem? After rethinking the scenario, perhaps they did incorrectly view the 
situation. Are they monitoring their behaviour based on experience, or did they consider 
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additional information retrieved from their elaborated schemata and just change their opinion? 
This situation is difficult to classify in the assessment of skills hierarchy. 

Royer, Cicero, and Carlos (1993) offered a strategy to assess cognitive development using 
the theory that Glaser et al., (1985) proposed and linked it to Anderson’s (1983) ACT* model for 
acquisition of knowledge. These researchers suggest that knowledge acquisition runs parallel to 
the assessment of clinical reasoning skills along a continuum. The authors of the present study 
propose that knowledge acquisition, assessment of these skills, and medical education could be 
overlaid on the same continuum. 

When clinician instructors design items they are attempting to discriminate between 
competent reasoning and incompetent reasoning. They must create alternatives that competently 
reasoning students would know to be incorrect, but that other students might be attracted to. 
Similarly, when they choose from a bank, they should be able to identify which options 
competently reasoning students should eliminate as obviously incorrect. They should know which 
options are moderately wrong, an indication of partial knowledge, and which option is the correct 
response, the keyed option. For clinicians to identify options within these categories, they must 
be further along the knowledge acquisition continuum than students (Anderson, 1983), but if they 
are further along the continuum they may not appreciate the complexity and possible ambiguity 
that would be apparent to competently reasoning students. When clinicians examine distractors, 
can they provide a rationale for eliminating a distractor as incorrect that coincided with the 
students’ reasons? Did the clinicians’ explanations of student reasoning parallel the students’ 
rationales for choosing the keyed alternative? If clinicians successfully performed these two 
tasks, it would verify their ability to move down the continuum to think from a student’s 
perspective. 
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Method 



Selection of the Subjects 

Each clinician was a physician: (1) licensed to practice medicine, (2) with more than five 
years of clinical practice experience, (3) with a minimum of five years of current experience in 
medical education, (4) who had contact with medical students in his/her daily practice, and (5) 
who had instructional experience. Of the 12 physicians in this study, there were four from family 
practice, two each from general internal medicine, infectious diseases and pulmonary medicine, 
and one each from haematology and gastroenterology. Forty medical students who had just 
completed a medicine clinical rotation in their third year of medical school were asked by Skakun 
(1994) to think aloud as they responded to 30 multiple-choice items selected from an item bank 
in the Department of Internal Medicine. Results from the four items are reported in Table 1. 

Data Collection 

All of the clinicians who consented to be in the study were sent preliminary materials to 
review. The information package included an overview of the interview, a consent form, and an 
example of an item being solved using a think aloud strategy. Clinicians were first asked to think 
aloud as they solved the item themselves, and then asked to go back and offer reasons students 
would use to eliminate distractors. Each clinician instructor was interviewed independently 
without interruption and completed in one session. Because the present study was part of a 
larger study on standard setting, the total time required to gather data from each clinician 
instructor was between two and three and one-half hours. Competent reasoning was 
operationally defined as choosing the keyed response for this study. 

All interviews were recorded on audiotapes and field notes were taken. Transcriptions were 
done immediately after the interviews. After each interview was transcribed, it was verified by 



Clinicians’ Explanations of Student Reasoning While Solving Multiple-Choice Items 



March 19, 1997 



9 

listening to the audiotape while reading the protocol for accuracy (Triska, 1996). Students 
transcripts (i.e., protocols), obtained from Skakun (1994) were used to compare clinician 
reasoning and student reasoning for the same MC items. 

Analysis 

Once competence was established, identification was removed from all of the competently 
reasoning clinician and student protocols. The clinician and student protocols were randomly 
interspersed for each item and the items were sorted by the researcher into two categories, 
novice or expert, based on the reasoning displayed in the protocol (Royer, et. al., 1993). The 
purpose of the dichotomous sort was to determine if clinicians using expert reasoning could be 
separated from student novice reasoning (Table 2). In the discussion that follows, the terms 
expert and novice clinicians and students refer to students and clinicians who demonstrated 
expert and novice reasoning on a particular item. 

Since the focus of this inquiry was on the behaviour of clinicians, the analysis centred on 
whether the reasons clinicians gave agreed with the reasons students gave for eliminating 
distractors. To determine if the clinicians’ insights into the students’ reasoning processes were 
accurate or inaccurate, a content analysis of each protocol was done. 

For the distractor analysis, the clinicians’ reasons for eliminating each distractor were 
categorized and coded. Students’ protocols were examined to see if their reasons paralleled the 
clinicians’ reasons. The results of the categorization and coding of the clinicians’ reasons for 
elimination and students’ protocols were verified by a clinician. The interrater agreement, using 
proportions, for the distractor analysis ranged from 75% to 92%. 

Due to the qualitative nature of the analysis of the clinicians’ and students’ protocols, and 
due to the spontaneous nature of the clinicians’ explanations, the criterion used to determine if 
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clinicians could reason like students was set at a fairly low level. When clinicians did not identify 
more than half of the reasons students used to eliminate each distractor as incorrect the data 
were taken as not supporting the theory. The proportion of agreement between the two groups 
was calculated by using the formula: 

Number of Reasons Listed by Clinicians and Used by Students 

Proportion of Agreement = 

Number of Reasons the Students Listed by Clinicians + Other 
Reasons Listed by Students 

Skakun (1994) recorded only the final justification for eliminating a distractor. In the few 
cases where they gave more than one reason, the final one was the dominant factor in their 
decision. 

For the keyed response analysis, clinician protocols were examined to determine how a 
clinician thought a student would choose the keyed alternative. The clinicians’ reasons were 
categorized into four broad areas. An atomistic approach involved solving the item by examining 
each piece of information and using facts, descriptions, and recalling definitions to arrive at the 
correct response. In a holistic approach, the individual uses concepts and principles to solve the 
item. Other clinicians stated what they expected students to know to solve the item. The final 
group of clinicians gave no explanation of how a student solved the item. 

Students’ item solving strategies were compared to the clinicians’ explanations. The result of 
the content analysis was verified by another rater (Tables 3, 4, 5, 6). The interrater agreement 
for the clinicians’ perceptions of how students reasoned ranged from 54% to 80% (mean=67%). 
The interrater agreement for how students reasoned ranged from 60% to 77% (mean=69%). 
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Results 

Clinicians were never specifically asked to disclose why a student would reject an alternative 
rather, they were asked to indicate which alternatives students should know to be incorrect. The 
results of the distractor analysis showed that clinicians do not usually offer a reason to eliminate 
an alternative as incorrect spontaneously. This clinician behaviour could be attributed to the 
clinicians’ position on the knowledge continuum, that is, the individual’s automaticity and 
efficiency of processing information. It appeared that the expert clinicians’ knowledge structures 
were so well formed that the declarative knowledge was not proximal and they were using 
procedural knowledge and composed knowledge only. Where justifications were presented, 
clinicians did so without prompting. 

A summary of the results of the proportion of agreement of reasons for eliminating 
alternatives between the clinicians and students showed that as a group, clinicians did not identify 
reasons students used to eliminate distractors for 12 of the 16 distractors (75%) for the four 
items (Table 7). Contrary to expectations, the expertly reasoning clinicians however were more 
successful in identifying reasons students used to eliminate distractors as compared the novice 
clinicians. However, in a general sense, with both the expert and novice clinicians, neither group 
effectively viewed the item from a student’s levels of knowledge according to the ACT theory 
(Anderson, 1983). 

Clinicians captured all of the reasons students stated for only three distractors (19%). 
Clinicians did not identify any of the students’ reasons for eliminating one distractor. This 
pattern of judgements suggests that the clinicians’ decisions on which alternatives are implausible 
or plausible may depend on the item’s content, the clinicians’ domain knowledge, and the 
clinicians’ ability to view the item from a student’s perspective. It may be that clinicians are 
unsure of justifications students use to eliminate alternatives, although this was not specifically 
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probed in the study. Also, it may be the case that if clinician instructors cannot identify students’ 
probable misconceptions, the standard of performance demanded by the item might be at a level 
inconsistent with the students’ knowledge levels at this point in their medical training. 

The results of the clinicians’ perceptions of how students reasoned to choose the keyed 
alternative for the four items varied (Table 8). The novice reasoning clinicians thought students 
would use a variety of strategies to reason to the keyed response. Novice clinicians gave more 
explanations that were parallel to students’ solutions whereas expert clinicians described students’ 
solutions in more general terms (i.e., expectations). This finding suggests that novice clinicians 
were more likely to move down the knowledge continuum to the students position on the ACT* 
continuum. This group of clinicians’ judgements, the competent novice reasoning clinicians, 
would be in concert with the students cognitive development. 

None of the expertly reasoning clinicians said that students would solve any of the items 
using an atomistic approach. These clinicians thought that students would use a higher level of 
knowledge to reason to the keyed response, that is, students would use a conceptual, principled 
approach to solving the item. The expert reasoning clinicians tended to state their expectation of 
the students’ knowledge levels, and frequently did not explain how students would reason to 
select the keyed response. This group of competently reasoning clinicians did not move to the 
students positions on the ACT* continuum. Clinicians in this category might select items for an 
examination beyond the capabilities of the students at a given level. 

The findings of the clinicians’ perceptions of students’ reasoning for solving items revealed 
two prominent issues. First, expertly reasoning clinicians were more successful in identifying the 
students’ justifications for eliminating distractors than novice reasoning clinicians. There were 
fewer novice reasoning clinicians in this study, therefore a note of caution in interpreting the 
results is needed. Second, novice reasoning clinicians were more successful than expert reasoning 
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clinicians in explaining how students reasoned to select the keyed response. These results suggest 
that the competently reasoning expert and novice clinicians’ decisions complimented each other 
in identifying the distractor elimination and perceptions of how students solved items. 

In view of the results of this study, further research should be conducted in linking cognitive 
psychology with: (1) measurement, and (2) standard setting. In measurement, the effect of 
cognitive psychology training and item writing for clinicians on item analysis and distractor 
analysis should be investigated. Distractor analyses should be done to compare the effect of 
clinician responses before and after prompting to explain misconceptions in distractors, then 
compared with reasons students gave to eliminate distractors. Item statistics should be compared 
before and after clinicians were given instruction on the students’ position on the knowledge 
continuum. 

In setting standards of performance, clinicians require an understanding of certain aspects of 
cognition, particularly knowledge acquisition and the position of clinicians and students on the 
knowledge continuum. Clinicians must be made aware of their level of automaticity, efficiency of 
processing information, and highly proceduralized and composed knowledge structures, and how 
they differ from the students. Without this awareness, clinicians may set unrealistic standards for 
students at a particular level of training. 

Standard setting committee members should be aware of the role of the experts’ and novices’ 
knowledge structures and decision-making characteristics to use both effectively. This study 
showed that experts’ judgements had higher average consistency than the novices’ judgements, 
however the novices were sometimes more successful in viewing an item from a student’s 
perspective. Having both experts and novices reach consensus supports the reproducibility and 
validity of the standard setting judgements. 

The time has come to integrate cognitive psychology with psychometrics in order to produce 
valid MC items and reproducible standard setting judgements of performance. 




Clinicians’ Explanations of Student Reasoning While Solving Multiple-Choice Items 



March 19, 1997 



14 



References 

Anderson, J. R. (1983). The architecture of cognition. Cambridge, MA: Harvard University 
Press. 

Anderson, J. R. (1993). Problem solving and learning. American Psychologist, 48( 1), 35-44. 

Anderson, J. R. (1995). Cognitive psychology and its implications (4th ed.) New York: W. H. 
Freeman. 

Glaser, R., Lesgold, A., & Lajoie, S. (1985). Toward a cognitive theory for the measurement 
of achievement. In R. R. Ronning, J. Glover, J. C. Conoley, & J. C. Witt (Eds.) The influence of 
psychology on testing and measurement (pp. 41-85). Hilldale, NJ: Lawrence Erlbaum Associates 
Publishers. 

Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational & 
Psychological Measurement, 14, 3-19. 

Royer, JM, Cicero, CA, & Carlo, MS. (1993). Techniques and procedures for assessing 
cognitive skills. Review of Educational Research, 63, 201-243. 

Skakun, E. N. (1994). Strategy choices in responding to multiple items. Unpublished doctoral 
dissertation. University of Alberta, Edmonton, Alberta, Canada. 

Triska, O. H. (1996). The validity of clinicians’ judgements in standard setting. Unpublished 
doctoral dissertation. University of Alberta, Edmonton Alberta, Canada. 




Clinicians’ Explanations of Student Reasoning While Solving Multiple-Choice Items 



March 19, 1997 



15 



15 



Table 1 

Multiple Choice Item Used in the Study 



ITEM 1 

A previously healthy 27 year-old female is 
suddenly seized with pleuritic pain in the left 
chest and shortness of breath. The most 
likely cause is: 

1. mycoplasma pneumonia. 

* 2. spontaneous pneumothorax. 

3. pulmonary embolism. 

4. acute pericarditis. 

5. pleurodynia. 



ITEM 2 

A 28 year-old environmental activist has a 
history of having had pneumonia four times 
in the past twenty years. She has had a 
productive cough "all her life" which is worse 
in the winter. Physical examination reveals 
dullness, diminished breath sounds and 
numerous crepitations below T3 bilaterally. 
Her fingers are clubbed. The most likely 
diagnosis is: 

1. hypogammaglobulinemia. 

2. congenital heart disease. 

3. bronchiolitis obliterans. 

* 4. bronchiectasis. 

5. cystic fibrosis. 



ITEM 3 

A 56 year-old man presents with a month 
history of intermittent right facial pain. On 
examination he is found to have a 
diminished right corneal reflex and a slight 
hearing defect on the same side. The 
diagnosis is: 

1. right cerebral tumour. 

2. trigeminal neuralgia. 

3. otitis media. 

* 4. acoustic neuroma. 

5. multiple sclerosis. 

ITEM 4 

A 24 year-old airline flight attendant 
complains of feeling tired and losing weight 
in spite of a good appetite. For the past 
year she has noticed voluminous, pale, foul- 
smelling stools. She recalls being told of 
having bowel difficulty in early childhood 
and of being fed a diet largely consisting of 
bananas. Radiological examination discloses 
an abnormal small bowel follow through. 
Biochemical analysis of the stool shows an 
increased amount of fat. The blood picture 
shows anemia. Which of the following diets 
would you select for this patient? 

* 1. Gluten free 

2. Lactose free 

3. Low fat 

4. Low residue 

5. High residue 
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Table 2 



Item 1: Clinicians’ and Students’ Problem Solving Strategies 



Group 


Expert Reasoners 


Novice Reasoners 


Total 


Clinicians 


8 


3 


11 of 12 


Students 


16 


5 


21 of 40 



Item 2: Clinicians’ and Students’ Problem Solving Strategies 



Group 


Expert Reasoners 


Novice Reasoners 


Total 


Clinicians 


3 


2 


5 of 12 


Students 


14 


8 


14 of 40 



Item 3: Clinicians’ and Students’ Problem Solving Strategies 



Group 


Expert Reasoners 


Novice Reasoners 


Total 


Clinicians 


3 


5 


8 of 12 


Students 


9 


9 


18 of 40 



Item 4: Clinicians’ and Students’ Problem Solving Strategies 



Group 


Expert Reasoners 


Novice Reasoners 


Total 


Clinicians 


9 


2 


11 of 12 


Students 


19 


13 


19 of 40 
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Table 3 



Item 1: Summary of Reasons for Eliminating Distractors 



Stem: A previously healthy 27 year-old female is suddenly seized with pleuritic pain in the left chest and shortness of 
breath. The most likely cause is: 

Keyed Option: 2. Spontaneous pneumothorax. 



Alternative 1 . Mycoplasma pneumonia. 



Reasons for elimination. 



Eliminated by 
Clinicians Students 



N 



** 



1. Mycoplasma pneumonia is not characterized as a catastrophic event. 




cr** 


X 


2. Additional symptoms are present with mycoplasma pneumonia. 


X 


X 


X 


3. The number of pneumonias is unusual. 


X 


o 


X 


4. The presentation does not fit with mycoplasma pneumonia. 


o 


o 


X 


Alternative 3. Pulmonary embolism. 


1. No risk factors are stated. 


X 


o 


X 


2. The history does not indicate a pulmonary embolism. 


o 


o 


X 


3. The age is inconsistent with pulmonary embolism. 


o 


o 


X 


Alternative 4. Acute pericarditis. 


1. Suddenly seized is inconsistent with acute pericarditis. 

2. Acute pericarditis is quite unlikely due to the absence of 


X 


o 


X 


symptoms if a viral infection or bacterial infection. 


o 


X 


o 


3. Given the framework of this question, this is incorrect. 


o 


o 


X 


4. The age is inconsistent with acute pericarditis. 


o 


o 


X 



Alternative 5. Pleurodynia. 



Can be associated with a prodrome or premonitary symptoms, 
such as a viral infection or fever. 


X 


o 


o 


Pleurodynia is not associated with a sudden onset of pain. 


X 


o 


X 


Pleurodynia is not associated with shortness of breath. 


X 


o 


o 


Pleurodynia can present like this. 


X 


o 


o 


Do not know what pleurodynia is. 


o 


o 


X 



Note: 



E* = Expert reasoning clinicians; 

*** 

X = Reason given; 



N = Novice reasoning clinicians. 

a): a|c s|e a|c 

O = No reason given 
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Table 4 

Item 2: Summary of Reasons for Eliminating Distractors 



Stem: A 28 year-old environmental activist has a history of having had pneumonia four times in the past 

twenty years. She has had a productive cough "all her life" which is worse in the winter. Physical examination 
reveals dullness, diminished breath sounds and numerous crepitations below T3 bilaterally. Her fingers are 
clubbed. The most likely diagnosis is: 

Keyed Option . 4. Bronchiectasis. 



Alternative 1 . Hypogammaglobulinemia. 



Reasons for elimination. 

1. Skin lesion infections are often associated with hypogammaglob. 

2. Does not present with these physical findings. 

3. Recurrent infections are present with hypogammaglobulinemia. 



Eliminated by 
Clinicians Students 
E N 



X 


o 


o 


X 


o 


X 


o 


o 


X 



Alternative 2 . Congenital heart disease. 



1. Additional findings would be present. 

2. Congenital heart disease presents with shortness of breath, 


X 


o 


X 


cyanosis, and clubbing. 


X 


o 


X 


3. History indicative of pulmonary problems. 


X 


o 


X 


Alternative 3. Bronchiolitis obliterans. 


1. A disease not known by the students. 

2. Students could be confused and think that this on one of the 


X 


o 


X 


manifestations of bronchiolitis obliterans. 


o 


o 


o 


3. This is the wrong presentation for bronchiolitis obliterans. 


o 


o 


X 


4. This disease is more of an autoimmune problem. 


o 


o 


X 


5. Not enough information to diagnose bronchiolitis obliterans. 


o 


o 


X 


6. Bronchiolitis obliterans progresses faster. 


o 


o 


X 



Alternative 5 . Cystic fibrosis. 



In comparing bronchiectasis with cystic fibrosis, bronchiectasis 
would be much more likely. 


X 


o 


o 


Age is inconsistent with cystic fibrosis. 


o 


o 


X 


Cystic fibrosis affect the upper respiratory tract. 


o 


o 


X 


More problems are associated cystic fibrosis. 


o 


o 


X 


Not information to diagnose cystic fibrosis. 


o 


o 


X 



Note: 



E = Expert reasoning clinicians; 

V s * 4 -p \ 1 

X = Reason given; 



N = Novice reasoning clinicians. 

**** . ° 

O = No reason given. 
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Table 5 

Item 3: Summary of Reasons for Eliminating Distractors 



Stem: A 56 year-old man presents with a month history of intermittent right facial pain. On 

examination he is found to have a diminished right corneal reflex and a slight hearing defect on the same 
side. The diagnosis is: 

Keyed Option . 4. acoustic neuroma. 



Alternative 1 . Right cerebral tumour. 



Eliminated by 







Clinician 


Student 




Reasons for elimination. 


E’ 


N 




1 . 


Symptoms should be on the opposite side. 


X*’* 


X 


o’" 


2. 


Doesn’t explain the pain and diminished corneal reflex 


X 


o 


X 


3. 


Doesn’t cause right facial pain. 


o 


X 


o 


4. 


Symptoms suggest cranial nerve involvement, not cerebellum. 


o 


o 


X 


5. 


Symptoms suggest that the problem is not in the rt. cerebellar area. 


o 


o 


X 


5. 


The symptoms would be more localized. 


o 


o 


X 


6. 


The symptoms would be more generalized. 


o 


o 


X 


7. 


Do not know how this relates to the scenario. 


o 


o 


X 


Alternative 2. Trigeminal neuralgia. 








i. 


Physical findings are not totally explained by trigeminal neuralgia. 


X 


X 


X 


2. 


Does not have a diminished corneal reflex. 


X 


o 


o 


Alternative 3. Otitis media. 








i. 


Doesn’t explain the corneal reflex involvement. 


o 


X 


X 


2. 


Not a probable presentation of otitis media. 


X 


o 


X 


3. 


No prior history of chronic ear inflammation. 


X 


o 


X 


Alternative 5. Multiple sclerosis. 








i. 


Multiple sclerosis is more central. 


X 


o 


X 


2. 


Multiple sclerosis does not produce pain. 


X 


o 


o 


3. 


Multiple sclerosis is more specific. 


X 


o 


X 


4. 


Not a typical presentation of multiple sclerosis. 


o 


o 


X 


5. 


Optic nerve involvement is not present. 


o 


o 


X 



Note: E* = Expert reasoning clinicians; 

X = Reason given; 



N** = Novice reasoning clinicians. 

O**** = No reason given. 
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Table 6 

Item 4: Summary of Reasons for Eliminating Distractors 



Stem: A 24 year-old airline flight attendant complains of feeling tired and losing weight in spite of a good 

appetite. For the past year she has noticed voluminous, pale, foul-smelling stools. She recalls being told of 
having bowel difficulty in early childhood and of being fed a diet consisting largely of bananas. Radiological 
examination discloses an abnormal small bowel follow through. Biochemical analysis of the stool shows an 
increased amount of fat. The blood picture shows anemia. Which of the following diets would you select for 
this patient? 



Keyed Option L Gluten free 



Alternative 2. Lactose Free. 



Eliminated by 
Clinicians Students 



Reasons for elimination. 


E* 


n" 




Lactose intolerance does not result in an abnormal 








small bowel follow though. 




X 


o’ 


Excess fat in the stool not associated with lactose intolerance. 


X 


o 


o 


Weight loss not associated with lactose intolerance. 


X 


o 


o 


Fatigue not associated with lactose intolerance. 


X 


o 


o 


Anemia not associated with lactose intolerance. 


X 


o 


o 


This scenario is not a presentation of lactose intolerance. 


o 


o 


X 


Diarrhea is not present. 


o 


o 


X 



Alternative 3. Low Fat. 



1 . 


Not going to help. 


X 


o 


X 


2. 


This is not a malabsorptive diagnosis. 


X 


o 


X 


3. 


No pancreatic or gall bladder disease is present. 


o 


o 


X 


4. 


The mucosa is not damaged. 


o 


o 


X 


5. 


This diet will not address the underlying pathophysiology. 


o 


o 


X 


6. 


Fat is needed in the diet. 


o 


o 


X 


7. 


Do not know what this will do for the problem. 


o 


o 


X 



Alternative 4. Low Residue. 



1. Not going to help. 


X 


o 


X 


2. Not a large bowel problem. 


o 


o 


X 


3. Do not know what this will do for the problem. 


o 


o 


X 


Alternative 5. High residue. 


1. Not going to help. 


X 


o 


X 


2. Will make their voluminous stools more voluminous. 


X 


o 


o 


3. Not a large bowel problem. 


o 


o 


X 


4. No diarrhea is present 


o 


o 


X 


5. Do not know what this will do for the problem. 


o 


o 


X 



$ 

Note: E = Expert reasoning clinicians; 

X = Reason given; 



N = Novice reasoning clinicians. 
O**** = No reason given 
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Table 7 

Summary of the Agreement of Distractors Elimination Between the Clinicians and Students 



Dis tractor 



Item No. 


1st 


2nd 


3rd 


4th 


1 


0.75 


0.33* 


0.50* 


0.50* 


2 


0.50* 


1.00 


0.20* 


0.00* 


3 


0.17* 


1.00 


1.00 


0.50* 


4 


0.29* 


0.28* 


0.33* 


0.25* 



Note: *Clinicians did not successfully identify >0.50 of the reasons students gave for 

eliminating a distractor. 



Table 8 



Summary of the Clinicians’ Perceptions of the Students’ Reasoning to Solve Items 



Novice Clinicians 



Expert Clinicians 



Item 


A 


H 


E 


NE 


A 


H 


E 


NE 


1 


O* 


X** 


O 


O 


O 


X 


X 


X 


2 


X 


X 


O 


O 


O 


o 


X 


o 


3 


X 


X 


X 


o 


O 


X 


X 


X 


4 


o 


o 


X 


X 


O 


X 


X 


X 



Note: O* = No clinicians; X** = Clinicians in this category. 
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